Imaging apparatus for generating HDR image from images captured at different viewpoints and method for controlling imaging apparatus

ABSTRACT

An imaging apparatus includes an image sensor including a microlens array between an imaging optical system and a plurality of pixels and configured to receive a light flux from each of microlenses in the microlens array at the plurality of pixels to output an image signal, and a generation unit configured to select, among a plurality of pixel signals obtained at different viewpoints corresponding to a subject in the image signal from the image sensor, the pixel signal based on the brightness of the pixel signal serving as a reference and positions of the pixels at which the plurality of pixel signals are output relative to each of the microlenses, to generate an output image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an imaging apparatus capable ofsimultaneously acquiring images from different viewpoints and ofgenerating a high dynamic range image based on the acquired images; italso relates to a method for controlling the imaging apparatus.

2. Description of the Related Art

Conventionally, Japanese Patent Application Laid-Open No. 2003-179819discusses, in order to generate an image having a high dynamic range(HDR) apparently wider than a dynamic range of an image, which can beacquired by performing imaging once, a technique for regularly arrangingpixels with different aperture ratios and generating the HDR image.

Japanese Patent Application Laid-Open No. 2013-072906 discusses aconfiguration of an imaging apparatus capable of acquiring imagescaptured at different viewpoints by performing imaging once, in which apair of subject images formed by light fluxes passing through differentpupil areas is acquired and the acquired pair of subject images is usedto detect a focus using correlation calculation.

However, there has been no reference to generation of an HDR image usingthe configuration of the imaging apparatus discussed in Japanese PatentApplication Laid-Open No. 2013-072906, i.e., an image sensor including aplurality of pupil division pixels assigned to one microlens.

SUMMARY OF THE INVENTION

The present invention is directed to an imaging apparatus including animage sensor including a plurality of pupil division pixels assigned toone microlens and capable of providing an HDR image, and a method forcontrolling the imaging apparatus.

According to an aspect of the present invention, an imaging apparatusincludes an image sensor including a microlens array between an imagingoptical system and a plurality of pixels and configured to receive alight flux from each of microlenses in the microlens array at theplurality of pixels to output an image signal, and a generation unitconfigured to select, among a plurality of pixel signals captured atdifferent viewpoints corresponding to a subject in the image signal fromthe image sensor, the pixel signal based on the brightness of the pixelsignal and positions of the pixels at which the plurality of pixelsignals are output relative to each of the microlenses, to generate anoutput image.

According to another aspect of the present invention, a method forcontrolling an imaging apparatus including an image sensor including amicrolens array between an imaging optical system and a plurality ofpixels and configured to receive a light flux from each of microlensesin the microlens array at the plurality of pixels to output an imagesignal, includes specifying a plurality of pixel signals obtained atdifferent viewpoints corresponding to a subject in the image signal fromthe image sensor, selecting the pixel signals based on the brightness ofthe pixel signals and positions of the plurality of pixels relative toeach of the microlenses, and generating an output image using theselected pixel signals

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a cross-sectional view and FIG. 1B is a functional blockdiagram of an imaging apparatus, according to a first exemplaryembodiment.

FIGS. 2A, 2B, and 2C illustrate an imaging optical system according tothe first exemplary embodiment.

FIGS. 3A, 3B, and 3C are respectively a flowchart and diagramsillustrating HDR synthesis processing according to the first exemplaryembodiment.

FIGS. 4A, 4B, 4C, 4D, 4E, and 4F illustrate effects of HDR synthesis.

FIGS. 5A and 5B are respectively a flowchart and a diagram illustratinga process performed to change an aperture by a operating a diaphragmaccording to a second exemplary embodiment.

FIG. 6 is a flowchart illustrating a process of generating an imagebased on images acquired simultaneously from different viewpoints,according to a third exemplary embodiment.

FIGS. 7A, 7B, and 7C are respectively a flowchart and diagramsillustrating a pixel addition process according to a fourth exemplaryembodiment.

FIGS. 8A, 8B, 8C, and 8D illustrate addition of pixel values and adistribution (variation) of luminances.

FIGS. 9A and 9B illustrate a case where pixels have been saturated.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings. Animaging apparatus according to a first exemplary embodiment of thepresent invention will be described below with reference to FIGS. 1A to5B.

FIG. 1A is a cross-sectional view taken parallel to optical axis 4 atthe center of a digital camera serving as an imaging apparatus 1 and alens 2. FIG. 1B is a functional block diagram mainly illustrating anelectrical configuration. In FIGS. 1A and 1B, components respectivelyassigned the same reference numerals correspond to each other.

In FIG. 1A, an imaging apparatus 1 includes an image sensor 6, a displaydevice 9, a quick return mechanism 14, and a finder 16. The image sensor6 is a single-plate color image sensor including a complementary metaloxide semiconductor (CMOS) sensor with a primary color filter, forexample. The primary color filter includes three types of color filtersof red (R), green (G), and blue (B) respectively having transmissionmain wavelength bands in the vicinity of 650 nm, 550 nm, and 450 nm. Thedisplay device 9 is a display medium such as a liquid crystal display.The quick return mechanism 14 has a mirror. This mirror is rotated, toselectively introduce a light flux incident via an imaging opticalsystem 3 to the optical finder 16 or the image sensor 6. A lens unit 2including the imaging lens 3′ forming the imaging optical system 3 canbe electrically connected via an electrical contact 11 to and detachablyattached to the imaging apparatus 1.

An image processing circuit 7 is a processing system for processing animage formed on an imaging plane of the image sensor 6 via the imagingoptical system 3. Microlenses (hereinafter also referred to as MLs) arearranged in a lattice shape in two dimensions, as viewed in an opticalaxis direction, between the image sensor 6 and the imaging lens 3′forming the imaging optical system 3 (on a surface of the image sensor 6according to the present exemplary embodiment), to form a so-calledmicrolens array (hereinafter also referred to as MLA). The MLA performsa pupil division function in the present exemplary embodiment. In thepresent exemplary embodiment, pixels (light receiving units) on theimage sensor 6 have the same color filter in each ML unit, arranged in aBayer array. However, the present invention is not limited to this. Thepixels may be in a Bayer array in pixel unit, like in a normal imagesensor. However, in this case, the pixels assigned to each of the MLsmay include an odd number of pixels arranged in each of the vertical andthe horizontal directions of a lattice corresponding to the ML. If thepixels are assigned in units of odd numbers, the pixels at correspondingviewpoints under the adjacent MLs have alternately different colorfilters. Details of a function and an arrangement of the MLA will bedescribed below with reference to FIGS. 2A to 2C. As described below, afocus evaluation value and an appropriate exposure value are obtainedfrom the image sensor 6. Thus, the imaging optical system 3 isappropriately adjusted based on the focus evaluation and exposurevalues, so that the image sensor 6 is exposed to subject light with anappropriate light amount while a subject image is formed on the imagesensor 6.

The image processing circuit 7 includes non-illustrated components, suchas an analog-to-digital (A/D) converter, a white balance circuit, agamma correction circuit, an interpolation calculation circuit, and acolor conversion circuit using a look-up table. The image processingcircuit 7 receives an electrical signal from the image sensor 6 and cangenerate image signals for display and recording using these circuits.The image processing circuit 7 can also include an image processingdevice (a parallax image generation unit, a luminance correction unit,an alignment unit, a pixel selection unit, an output image generationunit, etc.) serving as a principal part of the image processingapparatus in the present exemplary embodiment. In the present exemplaryembodiment, it is assumed that the elements are arranged in the camerasystem control circuit 5.

A memory control unit 8 includes a processing circuit required forrecording in addition to an actual storage unit. The memory control unit8 performs output while generating and storing an image to be output toa display unit 9. The memory control unit 8 compresses an image, amoving image, and sound (e.g., voice) using a previously determined orknown method.

A camera system control circuit 5 generates a timing signal or the likeduring imaging, and outputs the generated timing signal. The camerasystem control circuit 5 controls an imaging system, an image processingsystem, and a recording and reproduction system, in response to externaloperations. An operation detection unit 10 detects that a shutterrelease button (not illustrated) is pressed, and controls driving of theimage sensor 6, an operation of the image processing circuit 7, andcompression processing of the memory control unit 8, for example.Further, the camera system control circuit 5 controls a state of eachsegment of an information display device that displays information on aliquid crystal monitor using the display unit 9.

An operation for adjusting an optical system by a control system will bedescribed. The image processing circuit 7 is connected to the camerasystem control circuit 5. The camera system control circuit 5 obtains anappropriate focus position and diaphragm position and level of opening(diameter) based on the signal from the image sensor 6. The camerasystem control circuit 5 issues an instruction to a lens system controlcircuit 12 via the electrical contact 11. The lens system controlcircuit 12 appropriately controls a lens driving unit 13. Further, acamera shake detection sensor (not illustrated) is further connected tothe lens system control circuit 12. In a mode in which the camera shakecorrection is performed, the lens system control circuit 12appropriately controls a shake correction lens via the lens driving unit13 based on a signal of the camera shake detection sensor.

FIGS. 2A, 2B, and 2C illustrate a principal part of the imaging opticalsystem 3 according to the present exemplary embodiment. While thepresent invention is also applicable to another imaging optical system,the other imaging optical system will be described with reference toFIGS. 4A, 4B, 4C, 4D, 4E, and 4F. To apply the present invention,subject images from a plurality of viewpoints need to be acquired. Inthe present exemplary embodiment, an MLA is arranged in the vicinity ofan imaging plane of the imaging optical system 3 to acquire angularinformation while a plurality of pixels is associated with one of MLsconstituting the MLA.

FIG. 2A schematically illustrates a relationship between the imagesensor 6 and an MLA 20. FIG. 2B is a schematic diagram illustrating thepixels (squares) on the image sensor 6 and the MLA including microlenses 20, 21, 22, and 23. FIG. 2C indicates that pixels under the MLA20 are respectively associated with specific pupil areas.

As illustrated in FIG. 2A, the MLA 20 is provided on the image sensor 6,and a front-side principal point of the MLA 20 is arranged to be inclose proximity to the imaging plane of the imaging optical system 3.FIG. 2A illustrates states of the MLA 20, respectively, as viewed fromthe side and the front of the imaging apparatus 1. A lens of the MLA 20is arranged to cover the pixels on the image sensor 6 when viewed fromthe front of the imaging apparatus 1. While each of the MLs constitutingthe MLA 20 is drawn large to be easily seen in FIG. 2A, the ML isactually only approximately several times the size of the pixel (theactual size will be described with reference to FIG. 2B).

FIG. 2B is a partially enlarged view of the relationship between pixelsand the microlens array, as viewed from the front of the imagingapparatus 1. A lattice-shaped frame illustrated in FIG. 2B representsthe two-dimensional arrangement of each of the pixels on the imagesensor 6. On the other hand, circles 20, 21, 22, and 23 respectivelyrepresent the MLs constituting the MLA 20. As apparent from FIG. 2B, aplurality of pixels is assigned to one of the MLs. In an exampleillustrated in FIG. 2B, up to 25 pixels of 5 rows×5 columns are assignedto one ML. More specifically, the size of each of the MLs isapproximately 5 times the size of the pixel. However, the size of the MLdepends on a distance between the MLA 20 and the pixel on the imagesensor 6, and therefore the number of pixels in the sensor correspondingto each ML may vary.

FIG. 2C illustrates a cross-section of the image sensor 6 cut to includean ML optical axis and to make its longitudinal direction match ahorizontal direction of the figure. In FIG. 2C, the image sensor 6includes pixels (photoelectric conversion units) 20-a, 20-b, 20-c, 20-d,and 20-e. On the other hand, an exit pupil plane of the imaging opticalsystem 3 is illustrated in an upper part of FIG. 2C. Actually, while adirection of the exit pupil plane is a direction perpendicular to thepaper surface of FIG. 2C when made to match the direction of the imagesensor 6 illustrated in a lower part of FIG. 2C, its projectiondirection is changed for ease of description. In FIG. 2C,one-dimensional projection/signal processing will be described tosimplify the description. In the actual imaging apparatus 1, the onedimension can easily be expanded to two dimensions.

The pixels 20-a, 20-b, 20-c, 20-d, and 20-e illustrated in FIG. 2C arerespectively in corresponding positional relationships with pixels 20-a,20-b, 20-c, 20-d, and 20-e illustrated in FIG. 2B. As illustrated inFIG. 2C, each of the pixels is designed to be conjugate to a specificarea on the exit pupil plane of the imaging optical system 3 by the MLA20. In the example illustrated in FIG. 2C, the pixels 20-a, 20-b, 20-c,20-d, 20-e respectively correspond to areas 30-a, 30-b, 30-c, 30-d, and30-e of the exit pupil plane. More specifically, only a light flux,which has passed through the area 30-a on the exit pupil plane of theimaging optical system 3, is incident on the pixel 20-a. The same istrue for the other pixels 20-b to 20-e as each corresponds to pupilareas 30-b to 30-e. Further, pixels in the adjacent ML also correspondto the same areas on the exit pupil plane. That is, pixels 21-a, 21-b,21-c, and 21-d respectively correspond to the areas 30-a, 30-b, 30-c,and 30-d. As a result, angular information can be acquired from apositional relationship between the area on the exit pupil plane and thepixel on the image sensor 6.

To generate a plurality of images captured at different viewpoints fromthe optical system described in FIGS. 2A, 2B, and 2C, pixelsrespectively corresponding to the same pupil planes (pupil areas) of MLsare arranged while making use of the above-mentioned characteristics.Thus, the plurality of images at the different viewpoints is generated.

While in the above-mentioned digital camera capable of acquiring theplurality of images at the different viewpoints, movement of a focusposition after imaging and movement of the viewpoints have beenproposed, the present exemplary embodiment is directed to expanding adynamic range (high dynamic range synthesis, also referred to as HDRsynthesis). In the optical system according to the present exemplaryembodiment, the plurality of images at the different viewpoints cansimultaneously be acquired. Thus, even if a moving subject is imaged, amoving subject image can be unblurred. In the present exemplaryembodiment, the plurality of images at the different viewpoints(parallax images) is acquired in the above-mentioned configuration, andthe HDR synthesis is performed based on a difference in shading statefor each of the images.

A difference in shading state for each of the pixels under the ML (adifference in shading state among the images at the differentviewpoints), used in the present exemplary embodiment, will be describedbelow. As illustrated in FIGS. 2B and 2C, in the imaging optical system3, each of the pixels corresponds to a specific area of an exit pupil ofthe imaging lens 3′ by the MLA 20. A diaphragm for defining an F numberexists in the imaging optical system 3. Thus, a light beam does notreach the imaging optical system 3 from outside thereof. Morespecifically, the entire pixel area of each of the pixels 20-b, 20-c,and 20-d corresponds to the exit pupil. However, a part of each of thepixels 20-a, 20-e, and 20-n does not correspond to the exit pupil. Thus,light fluxes, which reach the pixels 20-a, 20-e, and 20-n, are morelimited by the imaging optical system than light fluxes which reach thepixels 20-b, 20-c, and 20-d. This is referred to as shading.

A ratio corresponding to the exit pupil in the pixel area is defined asan aperture ratio in the present specification. Since all the pixelareas of the pixels 20-b, 20-c, and 20-d correspond to the exit pupil,the aperture ratios of the pixels 20-b, 20-c, and 20-d are 100%. Theaperture ratios of the pixels 20-a, 20-e, and 20-n are less than 100%.Further, the aperture ratio of the pixel 20-n is less than the apertureratios of the pixels 20-a and 20-e.

The aperture ratio corresponds to an amount of light flux incident oneach of the pixels as compared to the amount of light flux arriving tothe pupil plane of the optical system, and it is based on thecorresponding pupil area. More specifically, the aperture ratiocorresponds to the brightness of each of the plurality of imagescaptured at the different viewpoints. In the example illustrated in FIG.2B, the brightness of the image at the viewpoint corresponding to thepixel 20-n (an outer edge of the pupil area) is approximatelyone-seventh the brightnesses of the images at the viewpointscorresponding to the pixels 20-b, 20-c, and 20-d. As described above,according to the present exemplary embodiment, an HDR image, which hasbeen subjected to dynamic range expansion, is generated using adifference in exposure due to the difference in aperture ratio.

When the subject is bright, the images at the viewpoints correspondingto the pixels 20-b, 20-c, and 20-d reach saturation (overexposure)faster than the image at the viewpoint corresponding to the pixel 20-n.Thus, one or more of pixels at four corners in a similar shading stateto the pixel 20-a and the pixel 20-e or the pixel 20-n are selected (asa reference) so that the images are not easily saturated (overexposed).

On the other hand, when the subject is dark, a light flux hardlypenetrates the pixel 20-n. Thus, the image at the viewpointcorresponding to the pixel 20-n is underexposed faster. One or more ofthe pixels 20-b, 20-c, and 20-d, or pixels around the center and with anaperture ratio of 100% are selected (as a reference) so that the imagesare not easily underexposed.

More specifically, the dynamic range can be expanded by respectivelyselecting the pixels with a relatively low aperture ratio and the pixelswith a relatively high aperture ratio in the bright subject and the darksubject.

A difference among the aperture ratios of the pixels under the ML,described above, is determined depending on positions of the pixelsrelative to the corresponding ML. Therefore, as actual pixel selectioncontrol, the pixels are selected based on the positions relative to thecorresponding ML, to select pixel values corresponding to the brightnessof the image at a target position.

Further, the shading state as illustrated in FIG. 2B is determineddepending on only a state of the imaging apparatus 1, and does notdepend on the subject. More specifically, the shading state can bepreviously known (predetermined). Therefore, brightness correction andblur correction can be performed, as described above referring to FIGS.3A and 3C. The shading state may be obtained from so-called numericalvalues in design, or may be measured in manufacturing test processes ofthe imaging apparatus 1 in a factory. Obtained shading information,together with data representing a gain for compensating for shortage inamount of light due to shading and an exposure level difference for thepixel with no shading, is recorded as a table data (look-up table) in amemory of the memory control unit 8.

FIG. 3A is a flowchart illustrating an imaging and recording operationof the imaging apparatus 1 according to the present exemplaryembodiment. Processes in the flow are respectively implemented by thecamera system control circuit 5 or by each unit in response to aninstruction from the camera system control circuit 5. An operation inthe flow is started when the operation detection unit 10 detectspower-on of the imaging apparatus and an instruction to start imaging tothe imaging apparatus 1.

In step S301, the camera system control circuit 5 acquires an imagesignal that has been captured by the image sensor 6 and processed by theimage processing circuit 7. A timing at which an image of the acquiredimage signal is not limited. An image signal of an image previouslycaptured may be stored in the memory of the memory control unit 8 andread out.

In step S302, the camera system control circuit 5 or the imageprocessing circuit 7 generates a plurality of images (parallax images)captured at different viewpoints from the image signal obtainedsimultaneously by the image sensor 6. As described with reference toFIGS. 2A-2C, light fluxes received by pixels at similar relativepositions in an ML pass through the same area on a pupil plane of theimaging optical system 3. Thus, an image at a certain viewpoint can begenerated by collecting the pixel signals at the similar relativepositions in the ML. If this processing is similarly performed for eachof the pixels under each ML in the MLA 20, a plurality of imagescaptured at different viewpoints can be generated.

In step S303, the image processing circuit 7 performs brightnesscorrection for adjusting a difference in brightness among the parallaximages due to a difference in shading state. As described above, theshading state greatly differs depending on the relative position of agiven pixel under the ML. Thus, the plurality of images captured at thedifferent viewpoints differs in brightness. Thus, to synthesize theplurality of images captured at the different viewpoints to generate anHDR image, the brightnesses of the plurality of images captured atdifferent viewpoints are adjusted in this step. In the present exemplaryembodiment, a gamma value for raising an output more with respect to aninput signal is totally applied to the image at the viewpoint withlarger shading, to correct the brightness in the gamma correction. Howmuch the image captured at each of the viewpoints has been shaded and towhat extent the brightness of the image is to be corrected aredetermined for each pixel under each ML from the table data stored inthe memory, as described above. The brightness of each of the parallaximages may be corrected by performing gain correction for uniformlyraising signal levels to a signal value of the parallax image. Whileonly a case where luminance correction is performed is described in theexample illustrated in FIG. 3A, the luminance correction is not anindispensable operation, as some pixels may not require luminancecorrection.

In step S304, the camera system control circuit 5 or the imageprocessing circuit 7 performs processing for calculating an image shiftamount for aligning the plurality of images captured at differentviewpoints. More specifically, one of the plurality of images capturedat the different viewpoints is used as a reference image, to search fora position, which matches the reference image, of the other image and tocalculate an image shift amount. Each of the plurality of images, whichis an output in step S303, is divided into a plurality of areas, andcalculation is performed for each of the areas. A so-called epipolarconstraint exists between the images captured at the differentviewpoints simultaneously acquired, and the search is performed only inthe constrained direction. If the images, which respectively passthrough the area 30-a and the area 30-e on the pupil plane illustratedin FIG. 2C, are compared with each other, a viewpoint position isshifted only in an X direction. Therefore, a search direction can belimited to the X direction. More specifically, an area image is cut outfrom the image corresponding to the area 30-a, and an image of the samesize is cut out at a position that is shifted in the X direction fromthe image corresponding to the area 30-e, and the images are comparedwith each other. In the comparison, most similar positions are foundusing a sum of absolute difference (SAD) between the images based onluminance values in the areas, to calculate a shift amount.

If the luminance correction is not performed in step S303, matching maybe performed using a characteristic amount, which is not affected bybrightness, represented by Scale-Invariant Feature Transform (SIFT). Inan operation for calculating a shift amount, the same physical amount asan evaluation value obtained in correlation calculation performed duringso-called phase different focusing is observed. Thus, subject distanceinformation relating a subject can also be acquired.

An image shift amount between corresponding areas in the images capturedat the different viewpoints differs depending on at which subjectdistance the subject exists. The larger the subject distance is, i.e.,the farther the subject exists, the larger the image shift amountbetween the images captured at the different viewpoints is, in the samefocus state. The subject at an equal distance from the imaging apparatus1 is not necessarily captured in a general image. Thus, in the presentexemplary embodiment, the image is divided into a plurality of areas andmost similar positions are respectively found in the areas to cope withthe fact that the subject distance of the subject differs for each ofthe areas.

In step S305, the camera system control circuit 5 or the imageprocessing circuit 7 performs alignment processing for aligning theplurality of images captured at the different viewpoints for each of theareas. In step S304, it is found which area of the other image matchesthe reference image in the same subject. Therefore, in step S305, pixelscorresponding to the same object are associated with one another.

In step S306, the camera system control circuit 5 or the imageprocessing circuit 7 performs HDR synthesis processing for selecting,from among a plurality of pixels composing the other image correspondingto each of the areas of the reference image, the pixels to expand thedynamic range, and synthesizing the selected pixels. In step S305 andthe preceding steps, the pixels corresponding to the same subject areassociated with one another between the images captured at the differentviewpoints. Thus, the pixels, the number of which is one or more and isthe number of the viewpoints or less, are selected depending on thebrightness (luminance value). In the present exemplary embodiment, thepixel corresponding to a viewpoint with a relatively low aperture ratio,which is not easily overexposed, is used as a reference, to select thepixels depending on the brightness. If the pixel with a too low apertureratio, e.g., an aperture ratio of 10% or less is used as a reference,the brightness may not be able to be correctly measured due to effectsof shading occurring in a portion having a large height of an image andeccentricity of a lens. Accordingly, in the present exemplaryembodiment, among the plurality of pixels corresponding to each of theMLs, the pixel at which the area of a light flux incident from the ML iswithin a predetermined range is used as a reference of the brightness.

When a pixel is selected according to the brightness of pixels composingan image at a viewpoint used as a reference, the pixels respectivelycorresponding to 5×5 viewpoints are classified into three groupsdepending on a difference in aperture ratio, as illustrated in FIG. 3B,for example, and the pixel in the corresponding group is selecteddepending on the brightness according to a graph illustrated in FIG. 3C.A method for classifying pixels corresponding to each of the MLsdepending on a difference in aperture ratio and their positions in anactually controlled manner is not limited to this. For example, Group 1and Group 2 in FIG. 3B may be integrated, to select a pixel P(2, 2)-P(4,4) in a central portion where there is no shading by an area other thanthe ML as a pixel having an intermediate luminance in which a mainsubject exists at a high probability.

Values of the selected pixels are weighted and added according to anaperture ratio and a pixel value of each of the pixels, to generatepixels composing an output image. More specifically, a correction widthof the pixel with a low aperture ratio by the brightness correction instep S303 is large so that noise also increases. Therefore, theweighting is decreased. If each of the pixels is underexposed oroverexposed, a weight is made to zero or small, to reduce an effect ofthe pixel. An image exceeding an expression range of the pixel valuesbefore processing is generated by weighting and adding the pixel values.Thus, the image is converted to match the dynamic range of the outputimage using an appropriate method. So-called dynamic range compressionis performed. The pixel values may be normalized and weighted and addedaccording to the number of pixels previously selected while the dynamicrange compression is simultaneously performed.

In step S307, an output image for display or recording is generated, anddisplayed or recorded by performing resizing or gamma correction for adisplay image or coding processing in a predetermined format such asJoint Photographic Experts Group (JPEG), to end the processing.

An effect of the HDR synthesis processing performed in step S306 in thepresent exemplary embodiment will be described in association with animage with reference to FIGS. 4A to 4F. FIG. 4A schematicallyillustrates an image at a viewpoint with a high aperture ratio. FIG. 4Bschematically illustrates an image at a viewpoint with a low apertureratio. FIGS. 4C and 4D are respectively luminance histogramscorresponding to FIGS. 4A and 4B, respectively. FIG. 4E schematicallyillustrates an output image, and FIG. 4F is a luminance histogramcorresponding to FIG. 4E. In FIGS. 4A and 4B, the images arerespectively captured at the different viewpoints. Therefore, positionson image planes change depending on a subject distance. FIG. 4 will bedescribed, assuming that the images have been aligned in FIG. 4.

In FIG. 4A, the image includes a bright area 401 a and a dark area 402a. In FIG. 4B, the image includes a bright area 401 b and a dark area402 b. In a composition illustrated in FIG. 4, the inside of a room andthe outside of a window are simultaneously imaged. Thus, a dynamic rangeof a subject is wide.

While FIG. 4C is a luminance histogram c FIG. 4A, a portion 403 is to bepaid attention to. In a portion 404 c, a luminance takes a maximumvalue, but a frequency is not zero. Therefore, pixels, which have beensaturated, exist. Pixels corresponding to the outside of the windowserving as the bright area 401 a in the image illustrated in FIG. 4A aresaturated. On the other hand, in a portion 403 c, a luminance takes aminimum value, but a frequency is substantially zero. Therefore, pixelsare not underexposed, and can be exposed under an appropriate conditionfor a subject inside the room including the dark area 402 a in theimage.

On the other hand, while FIG. 4D is a luminance histogram correspondingto FIG. 4B, a portion 404 is to be paid attention to. In a portion 403d, a luminance takes a minimum value, but a frequency is not zero.Therefore, pixels, which have been underexposed, exist. Pixelscorresponding to a shadow inside the room serving as the dark area 402 bin the image illustrated in FIG. 4B are underexposed. (An acquired imageis described in halftone to make a correspondence between the image andthe subject definite in FIG. 4, although it is correct to represent theimage in black when the pixels are underexposed). On the other hand, ina portion 404 d, a luminance takes a maximum value, but a frequency issubstantially zero. Therefore, pixels are not saturated, and can beexposed under an appropriate condition for the outside of the windowserving as the bright area 401 b in the image.

As apparent from the description with reference to FIGS. 4A to 4D, thebright area and the dark area of the subject may be respectivelyobtained from the image at the viewpoint with the low aperture ratioillustrated in FIG. 4B and the image at the viewpoint with the highaperture ratio illustrated in FIG. 4A. This processing is performed instep S306 in the flowchart illustrated in FIG. 3A. A value used for anoutput image is selected from an image at an appropriate viewpoint whileseeing an aperture ratio depending on a luminance of pixels. Further,the images are integrated while considering the aperture ratio so thatan output image illustrated in FIG. 4E can be obtained. A method for theintegration has been described above in step S307.

In FIG. 4E, an image includes a bright area 401 e and a dark area 402 e.Saturation and underexposure do not occur in any place. As a result,even in FIG. 4F serving as a luminance histogram corresponding to FIG.4E, respective frequencies in a portion 403 f where a luminance takes aminimum value and a portion 404 f where a luminance takes a maximumvalue are substantially zero.

As described above, according to the present exemplary embodiment, animage sensor, which receives a light flux from each of MLs in an MLAarranged between a imaging optical system and a plurality of pixels, atthe plurality of pixels can simultaneously acquire a plurality of imagescaptured at different viewpoints and can perform HDR synthesis based ona difference in shading state among the images.

In the present exemplary embodiment described above, to reduce an effectof shading corresponding to an image height not by an ML, well-knownshading correction may be performed during the image acquisition in stepS301 using a correction table previously storing a correction gaincorresponding to the image height.

In a second exemplary embodiment, a diaphragm in an imaging opticalsystem 3 is adjusted, to control aperture ratios of images captured atdifferent viewpoints and to control exposure of each of the images andan exposure level difference between the images.

FIG. 5A is a flowchart illustrating an imaging and recording operationof an imaging apparatus 1 according to the present exemplary embodiment.Each of processes in the flow is implemented by a camera system controlcircuit 5 or by each unit in response to an instruction from the camerasystem control circuit 5. In the flow, an operation detection unit 10detects power-on of the imaging apparatus or an instruction to startimaging to the imaging apparatus 1, to start the operations. In FIG. 5A,steps in which similar operations to those illustrated in FIG. 3A areperformed are assigned the same numbers, and description of portionsrespectively having similar functions is not repeated, and onlydifferent portions will be described.

In the processing illustrated in FIG. 5A, steps S501 and S502 areprovided before step S301.

In step S501, an image sensor 6 or a light metering sensor (notillustrated) performs light metering in response to the instruction fromthe camera system control circuit 5. Various methods can be used for thelight metering. However, a method for exposing the image sensor 6 priorto main imaging will be described below. A dynamic range of a subjectincluded in an image can be known by exposing the image sensor 6 underan appropriate condition (referred to as light metering exposure) toobserve its signal level.

In step S502, the diaphragm in the imaging optical system 3 is operatedin response to the instruction from the camera system control circuit 5.A lens driving unit 13 illustrated in FIG. 1B is used, to operate adiaphragm included in a lens unit 2. To what extent the diaphragm is tobe closed/opened will be described below.

As a result of the light metering exposure, if any of pixel existswithin an appropriate range, a luminance of the subject is flat(luminance unevenness is small), and a dynamic range need notparticularly be expanded. On the other hand, if the subject is saturated(overexposed) or underexposed (an area where electrons can hardly jumpout due to the exposure) exist, the dynamic range is insufficient, or anexposure condition is not appropriate. In the light metering exposure,when the saturation and the underexposure simultaneously occur within ascreen even if the exposure condition is appropriately adjusted, aluminance difference of the subject is larger than the dynamic range ofthe image sensor 6, and a dynamic range of the subject is worth beingexpanded. When the saturation and the underexposure occur even if theexposure condition is appropriately adjusted, as described above, theimage sensor 6 is exposed under more underexposure (small exposureamount) and overexposure (large exposure amount) conditions. As aresult, a dynamic range required to sufficiently represent the subject(defined as a dynamic range of the subject) can be known by searchingfor the condition that the saturation and the underexposure areeliminated.

To solve the saturation (overexposure) and the underexposure, the imagemay be able to be acquired under the condition that the exposure ischanged to include the dynamic range of the subject. In an apparatusthat performs exposure under different conditions in a time-divisionalmanner to synthesize images, an exposure condition is determinedaccording to the dynamic range of the subject as ±1 step, ±2 steps, or±3 steps, to acquire a plurality of images, and the image processing isperformed on the images. On the other hand, it is not easy in theapparatus according to the present exemplary embodiment to control anamount of exposure for each image, like in a time-divisional ormultiple-lens camera. An example illustrated in FIG. 5B is an example ofprocessing proposed by paying attention to that.

More specifically, the aperture of the optical system is reduced in stepS502 after light metering is performed in step S501, to change a ratioof an amount by which a light beam penetrates (a ratio of luminances ofa brightest viewpoint and a darkest viewpoint). Consequently, a ratio ofaperture ratios of images captured at different viewpoints changes, andan exposure condition including the dynamic range of the subject can beachieved. A relationship between the aperture size of the optical systemand the change in the ratio of the amount by which the light beampenetrates will be described with reference to FIG. 5B.

FIG. 5B illustrates a case where the diaphragm in the imaging opticalsystem 3 is operated from the state illustrated in FIGS. 1A-1B. An exitpupil corresponding to a state after the diaphragm is operated has anouter diameter 40.

As apparent from FIG. 5B, when the diaphragm is operated to rapidlydecrease an aperture ratio, shading increases starting at the outermostpixel. For example, FIG. 3B illustrates a diaphragm opened state with anouter diameter 20 where there is no shading caused by the diaphragm.From this state, an aperture of the diaphragm is only decreased byapproximately 5% so that an aperture ratio of a pixel 20-n is reduced tohalf or less. On the other hand, aperture ratios of pixels 20-b, 20-c,and 20-d near the center remain 100%. As a result, a difference betweenthe brightness of an image captured at a viewpoint corresponding to thepixel 20-n and the brightness of images at viewpoints corresponding tothe pixels 20-b, 20-c, and 20-d further increases (an exposure leveldifference is larger). The diaphragm is appropriately operated accordingto the dynamic range of the subject, as described above, with referenceto FIG. 3B, so that subject information can be grasped moreapproximately. As a method for determining the exposure condition, itcan be determined, as needed, which pixel is a properly exposed pixelbased on a light metering result.

As described above, according to the present exemplary embodiment, animage sensor, which receives a light flux from each of MLs in an MLAarranged between a imaging optical system and a plurality of pixels, cansimultaneously acquire a plurality of images captured at differentviewpoints, and perform HDR synthesis based on a difference in shadingstate among images. In the case, a diaphragm is controlled according toa light metering value so that an exposure level difference between theimages at the viewpoints can be controlled.

In a third exemplary embodiment, correction for adjusting a defocusedstate by recognizing that the defocused state differs depending on adifference in aperture ratio among a plurality of images captured atdifferent viewpoints and the defocused state changes with HDR synthesisprocessing for a composite image.

A flowchart illustrated in FIG. 6 will be described. In FIG. 6, steps inwhich similar operations to those illustrated in FIG. 3A are performedare assigned the same numbers, and description of portions respectivelyhaving similar functions is not repeated, and only different portionswill be described. FIG. 6 illustrates an example of processing byparticularly paying attention to the defocused state. Steps S601 andS602, described below, respectively pay attention to a difference indefocused state among a plurality of images captured at differentviewpoints and a change in defocused state caused by dynamic rangeexpansion processing.

In the processing illustrated in FIG. 6, steps S601 and S602 arerespectively provided between steps S303 and S304 and between steps S307and S308.

In step S601, aperture ratio correction is performed. The aperture ratiocorrection means that smoothing filtering for each area is performed ona plurality of images captured at different viewpoints based on ashading state of each of pixels. A difference in aperture ratiocorresponds to the fact that the plurality of images at the differentviewpoints has been captured using different F numbers. Further, if theF numbers become too large, a blur caused by diffraction occurs. Thus,the plurality of images captured at the different viewpoints differs inthe defocused state. Therefore, in step S601, the plurality of imagescaptured at the different viewpoints is subjected to smoothing filteringaccording to the shading state. When other images captured at differentviewpoints are subjected to smoothing filtering using an image at astarting point corresponding to the pixel with the highest apertureratio as a reference, similarity of the images can be increased. Thus,the images can easily and appropriately be made to match one another inthe subsequent step S304.

In step S602, smoothing processing is performed on the entire imageafter the HDR synthesis obtained in step S306. The smoothing processingmeans performing smoothing filtering. As described above, in the dynamicrange expansion processing in step S306, processing for selectingpixels, the number of which is one or more and is the number ofviewpoints or less depending on the brightness, is performed. When theprocessing is performed, a blur of the image is reduced than that in amethod for adding all images serving as a conventional method forgenerating an output image. A plurality of images captured at differentviewpoints corresponds to areas obtained by dividing a correspondingpupil area. Thus, if one of the images is paid attention to, the imagehas been captured by increasing the F number (reducing an aperture of adiaphragm). When one of the plurality of images captured at thedifferent viewpoints is paid attention to, the image has a large depthof focus and has a small blur. If pixels, the number of which is one ormore and is the number of viewpoints or less, are selected and added, animage, whose blur is smaller than that when all the images are added, isoutput. Therefore, an output image is subjected to smoothing filteringaccording to an output of an alignment unit in step S602, to obtain animage to which an appropriate blur has been added as an output. Asdescribed above, in step S304, the same physical amount as a phasedifference AF is observed. Therefore, a distance to the subject can beknown. In the present exemplary embodiment, in step S304, not only animage shift amount but also a subject distance on the side of an objectin each of the areas based on the image shift amount is calculated, anda blur is added to the image according to the subject distance. Morespecifically, a blur is not added to a focused area, and processing maybe performed so that the subject that is far on an image plane from afocus position becomes smoother.

Through this processing, a blur, which is similar to that in theconventional technique, is obtained, and an output image correspondingto the intension of a user can be obtained. Further, the extent of thesmoothing processing to be performed in step S602 may be adjusteddepending on setting of the output image. More specifically, if an imagehaving a small blur is desired to be obtained, the smoothing processingis weakly performed. If an image having a large blur is desired to beobtained, the smoothing processing may be strongly performed.

As described above, according to the present exemplary embodiment, animage sensor, which receives a light flux from each of MLs in an MLAarranged between an imaging optical system and a plurality of pixels,can simultaneously acquire images at different viewpoints and performHDR synthesis based on a difference in shading state among the images.In this case, correction for adjusting a defocused state is furtherperformed by recognizing that the plurality of images captured at thedifferent viewpoints differs in defocused state according to adifference in aperture ratio and the defocused state changes with theHDR synthesis processing for a composite image. Thus, the defocusedstate for each imaging or for each area can appropriately be adjusted.

In a fourth exemplary embodiment, pixels with the same aperture ratioare added to form a difference in exposure, thereby enabling synthesisprocessing in addition to the first exemplary embodiment.

FIG. 7A is a flowchart illustrating the processing according to thepresent exemplary embodiment. Each of steps is performed by a camerasystem control circuit 5 or by each unit according to an instructionfrom the camera system control circuit 5.

FIG. 7B schematically illustrates an operation of pixel addition. InFIG. 7A, blocks having similar functions to those illustrated in FIG. 1Bare assigned the same numbers.

In the present exemplary embodiment, when a signal of each of pixels issaturated, a range expansion effect is not obtained. Therefore, in lightmetering in step S501 illustrated in FIG. 7A, each of parallax images isset to be exposed under an underexposure condition so that the pixel ishardly saturated. In an imaging apparatus 1, an aperture and a shutterspeed are set using a program diagram so that a dynamic range of animage sensor 6 can effectively be used for an Ev value obtained from alight metering value. This condition is referred to as a proper exposurecondition. When the diaphragm is narrowed down or the shutter speed isincreased for the above-mentioned condition, an amount of light reachingthe image sensor 6 can be reduced. Such an exposure condition isreferred to as an underexposure condition. The parallax image is in adark state by being underexposed. However, the number of pixels to besaturated can be reduced. FIG. 7C illustrates an example of a programdiagram. (This diagram changes depending on the type of lens, forexample). According to this diagram, when an Ev value is 12, an exposurecondition is determined along a dot line 120. Imaging is performed usingan F number of 3.5 and a shutter speed of 1/320 s. When the parallaximage is underexposed, an exposure condition is determined along a dotline 121 by referring to a position of outline characters. As a result,imaging is performed using an F number of 4.0 and a shutter speed of1/500 s.

In step S701, addition and synthesis processing is performed. Luminancesof the pixels, which have been associated with one another in step S305,are added. A specific image of this operation has been illustrated inFIG. 7B, and will be described below.

In step S306 illustrated in FIG. 7A, an added signal obtained in stepS701 increases in level by the addition. Therefore, the added signal isconverted to match the dynamic range of an output image. An output pixelsignal is then selectively output, like in the other exemplaryembodiments, according to the brightness of a reference image for eacharea, to perform HDR synthesis.

An operation of a pixel addition unit will be described with referenceto FIG. 7B. In a graph illustrated in an upper part of FIG. 7B, avertical axis and a horizontal axis respectively correspond to aluminance and a pupil area. Bar graphs illustrated in the upper part ofFIG. 7B respectively represent signal levels of pixels that correspondto the same object and have passed through different pixel areas. Forexample, light fluxes, which have respectively passed through pixels20-a, 20-b, . . . , 20-e in FIG. 2C and correspond to the same object,are collected. In the optical system illustrated in FIG. 2C, when theobject and an MLA 20 are at conjugate positions by a lens unit 2 (in afocused state of a conventional camera), pixels under the same microlenscorrespond to the same object while the light rays thereto passingthrough different pupil areas. In such a case, signals of the pixels20-a, 20-b, . . . , 20-e are graphed. If the object and the MLA 20 arenot at the conjugate positions by the lens unit 2, the pixelscorresponding to light fluxes, which have passed through different pupilareas, and corresponding to the same object, exist under anothermicrolens. These relationships are associated with each other in stepsS304 and S305 illustrated in FIG. 7A. More specifically, an arrangementof the luminances of the pixels, which have been associated with oneanother in step S305, is the graph in the upper part of FIG. 7B. In anexample illustrated in FIG. 7B, the bar graphs respectively representsignal levels 201, 202, 203, and 204 that have passed through differentpupil areas.

In a graph in a lower part of FIG. 7B, a vertical axis corresponds to aluminance. The graph in the lower part of FIG. 7B is obtained by addingthe bar graphs in the upper part of FIG. 7B. An added signal 210 is avertical stack of the signal levels 201, 202, 203, and 204 of the pixelsobtained from the light fluxes that have passed through the differentpupil areas. As apparent from FIG. 7B, the added signal 210 has a signallevel exceeding a saturation level considered in a pixel unit. A rangeis expanded toward the higher luminance side than that in an imageobtained by collecting the pixel signals (corresponding to the parallaximage generated in step S302 illustrated in FIG. 7A).

A variation among signals and a situation where a subject having a lowluminance is acquired will be described below with reference to FIGS.8A, 8B, 8C, and 8D. FIGS. 8A and 8B correspond to FIG. 7B, where asignal of a subject having an intermediate luminance is acquired. FIGS.8C and 8D illustrate a state where a signal of a subject having a lowluminance is acquired, although similar in a notation method to FIGS. 8Aand 8B.

In FIGS. 8A and 8C, graphs with a luminance on the vertical axis and apupil area through which a light flux corresponding to a pixel haspassed and a frequency on the horizontal axis are written side by sidein a horizontal direction. FIGS. 8B and 8D are graphs respectivelyillustrating added signals corresponding to FIGS. 8A and 8C and with aluminance on the vertical axis.

In FIG. 8A, signal levels 201, 202, 203, and 204 of pixels respectivelycorresponding to light fluxes, which have passed through different pupilareas, and corresponding to the same object are not exactly identicaland are written while slightly shifting from one another. This indicatesthat the signal level of each of the pixels is acquired throughprobabilistic processes. The number of free electrons generated by aphoton, which has reached a photo diode (PD) surface of the image sensor6, has been grasped. At this time, the generation of the free electronsis controlled by the probabilistic processes, and is controlled by aPoisson distribution. More specifically, even if the same number ofphotons reaches the PD surface, the same number of free electrons is notnecessarily generated. The generation of the free electrons follows acertain probability distribution. If the number of events is large, thePoisson distribution approaches a normal distribution. A representationof this in the form of a probability density function is a graph 205 onthe right side of FIG. 8A. The graph 205 spreads with a certaindistribution centered around an average value determined by thebrightness of the subject.

The signal levels 201, 202, 203, and 204 of the pixels, whichrespectively correspond to the different pupil areas, are added, toobtain the graph in FIG. 8B. A signal level 210 is obtained by addingthe signal levels 201, 202, 203, and 204. A distribution is obtained byadding variances. This is represented as a graph 215. The graph 215 isillustrated as a distribution whose bottom is somewhat wider than thegraph 205 in the upper part. This makes a user feel as if a variationincreased. However, the signal level increases in proportion to anaddition number. On the other hand, a standard deviation obtained byadding variances increases only by its square root. Thus, the additioncan suppress the variation in terms of signal to noise ratio (S/N).

A subject having a low luminance will be described below with referenceto FIGS. 8C and 8D. Examples of FIGS. 8A, 8B, 8C, and 8D respectivelyillustrate situations where a signal level with the minimum resolutionof each of pixels is obtained with a probability of 50%. In the exampleillustrated in FIG. 8C, the signal levels with the minimum resolution ofthe pixels corresponding to light fluxes, which have passed throughpupil areas 221 and 223, are output, and the signal levels of the pixelscorresponding to light fluxes, which have passed through pupil areas 222and 224, are zero (i.e., the pixels are underexposed). A Poissondistribution in such a case is in a shape, whose bottom is pulled towardone side by adhering to an axis on one side, as illustrated in a graph225.

If the signals in this case are added, a signal 230 is obtained. Thesignal 230 has a level higher than the signal level with the minimumresolution by the addition, and also has a probability distribution 235.The larger the number of pupil areas to be added is, the lower theprobability that the probability distribution 235 reaches zero becomes.Thus, the occurrence of underexposure is reduced. As a result, a rangeis also expanded toward the low luminance side.

A behavior of signals during saturation will be described below withreference to FIGS. 9A and 9B. Notation methods illustrated in FIGS. 9Aand 9B are respectively similar to those illustrated in FIGS. 8A and 8B.However, FIGS. 9A and 9B illustrate an acquisition state of a signal ofa subject in which pixels are saturated.

If the subject as illustrated in FIGS. 9A and 9B is imaged, as describedreferring to FIGS. 8A, 8B, 8C, and 8D, signal levels of the pixels areto be distributed according to a probability distribution 245. However,the signal levels exceed a saturation level of one pixel. Thus, thesignal levels are cut off at a saturation level, like signal levels 241,242, 243, and 244 illustrated in FIG. 9A. If the signal levels areadded, as illustrated in FIG. 9B, a signal 250 obtained by adding the npixels has a signal level equal to a saturation level of the n pixels.Only a lower value than the level of an added signal obtained when thereis no saturation can be obtained, and an accurate brightness and colorcannot be expressed. In this case, an effect of range expansion has notbeen obtained.

Under a general imaging condition, no problem may occur if pixels, whichare saturated when exposed in so-called proper exposure, are representedas sufficiently bright luminance points. On the other hand, if a rangeexpansion effect according to the present exemplary embodiment isdesired to sufficiently be obtained and if a luminance point itself isdesired to accurately be acquired, like in a starry sky, the pixels areeffectively prevented from being saturated when exposed under anunderexposure condition. In the example of the operation illustrated inFIG. 7A, the exposure is performed under an underexposure condition instep S501.

The present invention is not limited to an apparatus having imaging asits principal objective, such as a digital camera. The present inventionis applicable to any apparatus, which contains an imaging apparatus orto which an imaging apparatus is externally connected, such as apersonal computer (of a lap top type, a desk top type, a tablet type,etc.) or a game machine. Therefore, an “imaging apparatus” in thepresent specification is intended to include any electronic apparatushaving an imaging function.

Other Embodiments

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application2013-267152 filed Dec. 25, 2013, and No. 2014-186868 filed Sep. 12,2014, which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. An imaging apparatus comprising: an image sensorincluding a microlens array between an imaging optical system and aplurality of pixels each of which is designed to be conjugate to aspecific area on an exit pupil of the imaging optical system by themicrolens array, and configured to receive a light flux from each ofmicrolenses in the microlens array at the plurality of pixels to outputan image signal; and a generation unit configured to select, among aplurality of pixel signals output from the image sensor, a pixel signalbased on brightness of the pixel signals and positions of pixelsrelative to each of the microlenses, to generate an output image.
 2. Theimaging apparatus according to claim 1, further comprising a parallaximage generation unit configured to generate a plurality of imagesignals obtained at different viewpoints from the image signal from theimage sensor, wherein the generation unit selects, among a plurality ofcorresponding pixel signals in the image signals of the plurality ofimages captured at the different viewpoints, the pixel signals based onthe brightness of the pixel signal in the image signal at the viewpointserving as a reference and the positions of the pixels at which theplurality of pixel signals is output relative to each of themicrolenses, and synthesizes the selected pixel signals.
 3. The imagingapparatus according to claim 1, further comprising a parallax imagegeneration unit configured to generate the image signals obtained at thedifferent viewpoints from the image signal from the image sensor,wherein the generation unit, according to the brightness of the pixelsignal, generates the output image using a pixel signal obtained byadding the plurality of corresponding pixel signals in the image signalsobtained at the plurality of viewpoints.
 4. The imaging apparatusaccording to claim 1, further comprising a luminance correction unitconfigured to correct, for the pixel signal serving as an output fromeach of the pixels on the image sensor, a luminance based on theposition of the pixel relative to each of the microlenses at the pixels.5. The imaging apparatus according to claim 2, further comprising afilter unit configured to subject the image signals of the plurality ofimages captured at the different viewpoints to smoothing filtering basedon the position of each of the pixels on the image sensor relative toeach of the microlenses.
 6. The imaging apparatus according to claim 1,further comprising a light metering unit configured to determine anexposure condition for imaging by the image sensor, wherein an exposurelevel difference between the pixels on the image sensor is determinedbased on an output of the light metering unit, to drive a diaphragmbased on the exposure level difference and to acquire the image signal.7. The imaging apparatus according to claim 2, further comprising analignment unit configured to align, for the image signals of theplurality of images captured at the different viewpoints, subjects inthe images, wherein the generation unit selects, among the plurality ofcorresponding pixel signals for each area in the plurality of imagescaptured at the different viewpoints aligned by the alignment unit, thepixel signals and synthesizes the selected pixel signals, to generate anoutput image.
 8. The imaging apparatus according to claim 7, wherein thealignment unit compares the image signal at the viewpoint serving as areference among the image signals of the plurality of images captured atthe different viewpoints with the image signal at the other viewpoint tocalculate an image shift amount for each area, and align the subject inthe area based on the image shift amount.
 9. The imaging apparatusaccording to claim 7, further comprising a second filter unit configuredto subject the image signal to smoothing filtering based on the imageshift amount calculated by the alignment unit.
 10. The imaging apparatusaccording to claim 1, wherein among the plurality of pixelscorresponding to each of the microlenses, the generation unit uses thepixel, in which the area of the light flux incident from the microlensis in a predetermined range, as a reference.
 11. The image sensoraccording to claim 1, wherein the microlens array is arranged in twodimensions as viewed from an optical axis of the imaging optical system,and the pixels on the image sensor are further arranged in twodimensions as viewed from the optical axis relative to each of themicrolenses.
 12. The imaging apparatus according to claim 1, whereinamong the pixels on the image sensor, the same color filter is used forthe pixels corresponding to the same microlenses in the microlens array.13. The image sensor according to claim 1, wherein the pixels on theimage sensor are arranged in a Bayer array in a pixel unit, and theplurality of pixels corresponding to each of the microlenses includes anodd number of pixels and an odd number of pixels respectively arrangedin both the length and the breadth of the microlens.
 14. A method forcontrolling an imaging apparatus including an image sensor, the imagesensor including a microlens array between an imaging optical system anda plurality of pixels each of which is designed to be conjugate to aspecific area on an exit pupil of the imaging optical system by themicrolens array, and being configured to receive a light flux from eachof microlenses in the microlens array at the plurality of pixels tooutput an image signal, the method comprising: specifying a plurality ofpixel signals output from the image sensor corresponding to a samesubject in the image signal; selecting the pixel signals based onbrightness of the pixel signals and positions of pixels relative to eachof the microlenses; and generating an output image using the selectedpixel signals.
 15. A non-transitory computer-readable storage mediumstoring a program for causing a computer to execute the method forcontrolling the imaging apparatus according to claim
 14. 16. An imagingapparatus comprising: an image sensor including a plurality of pixels; amicrolens array including a plurality of microlenses, each of themicrolens array arranged between an imaging optical system and the imagesensor which is designed to be conjugate to a specific area on an exitpupil of the imaging optical system by the microlens array, eachmicrolens projecting on a predetermined number of pixels of the imagesensor an image of a subject captured at different viewpoints; and animage generation unit configured to generate an output image of thesubject based on pixel signals output from the image sensor, wherein theimage sensor outputs a plurality of pixel signals corresponding to alight flux received through each microlens in the microlens array, andthe image generation unit selects among the plurality of pixels signalsoutput from the image sensor, a pixel signal based on brightness of thepixel signals and positions of pixels relative to each of themicrolenses.